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(54) Method and system for inputting gestures 

(57) The present invention relates to a system, 
method and medium for receiving and acting upon user 
input. In one embodiment, the user may only have ac- 
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cess to a limited input device, like a stylus. Using the 
present invention, a user is provided with intuitive re- 
sponses from the system based on inputs from the lim- 
ited input device. 
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[0009] A number of inputs with a stylus are possible 
For example, a user may tap a stylus, stroke the stylus 
hold the stylus at a given point, or hold then drag the 
stylus. Other Inputs and combinations are possible as 
noted by the above-identified applications, which are ex- 
pressly incorporated herein by reference. 
[001 0] As to a stroke operation, the system may drag 
an object, may maintain a current state or operation or 
being inking. Inking may include writing, drawing ' or 
adding annotations as described in greater detail in U 
S. Serial No. 60/212,825, filed June 21 , 2000, entitled 
"Methods for Classifying, Anchoring, and Transforming 
Ink Annotations" and incorporated by reference. 
[001 1] As to a tap operation, the system may add to 
existing writing, may select a new object, insert a cursor 
or insertion point, or may perform an action on a select- 
ed object. 

[0012] As to a hold operation, the system may simu- 
late a nght mouse button click or other definable event 
[0013] As to a hold and drag operation, the system 
may drag a selected object or perform other functions 
[0014] These and other features of the invention will 
be apparent upon consideration of the following detailed 
description of preferred embodiments. Although the in- 
vention has been defined using the appended claims 
these claims are exemplary in that the invention is in- 
tended to include the elements and steps described 
herein in any combination or subcombination. Accord- 
ingly, there are any number of alternative combinations 
fordefining the invention, which incorporate one ormore 
elements from the specification, including the descrip- 
tion, claims, and drawings, In various combinations or 
subcombinations. It will be apparent to those skilled In 
the relevant technology, in light of the present specifica- 
tion, that alternate combinations of aspects of the inven- 
tion, either alone or in combination with one or more el- 
ements or steps defined herein, may be utilized as mod- 
ifications or alterations of the invention or as part of the 
invention. It is intended that the written description of 
the invention contained herein covers all such modifica- 
tions and alterations. 



to implement various aspects of the invention. 
[001 7] Figure 2 is a plan view of a tablet computer and 
stylus that can be used in accordance with various as- 
pects of the present invention. 
[001 8] Figures 3-7 are flowcharts showing a variety of 
steps for interpreting a user's input in accordance with 
embodiments of the present invention. 

Detailed Description of Preferred Embodiments 



Brief Description of the Drawings 

[0015] The foregoing summary of the invention as 
well as the following detailed description of preferred 
embodiments, is better understood when read in con- 
junction with the accompanying drawings, which are in- 
cluded by way of example, and not by way of limitation 
with regard to the claimed invention. In the accompany- 
ing drawings, elements are labeled with three-digit ref- 
erence numbers, wherein the first digit of a reference 
number indicates the drawing number in which the ele- 
ment is first illustrated. The same reference number in 
different drawings refers to the same element. 
[0016] Figure 1 is a schematic diagram of a general- 
purpose digital computing environment that can be used 



[0019] The present invention may be more readily de- 
scnbed with reference to Figures 1-7. Figure 1 illustrates 
a schematic diagram of a conventional general-purpose 
digital computing environment that can be used to im- 
plement various aspects of the present invention. In Fig- 
ure 1 . a computer 1 00 includes a processing unit 1 1 0 a 
system memory 1 20, and a system bus 130 that couples 
vanous system components including the system mem- 
ory to the processing unit 1 1 0. The system bus 1 30 mav 
be any of several types of bus structures including a 

memo^ bus or memory controller, a peripheral bus, and 
a local bus using any of a variety of bus architectures 

L™!rA em mem0fy 120 includes read onl V memory 
(ROM) 140 and random access memory (RAM) 150 

[0020] A basic inpul/output system 160 (BIOS) con- 
taining the basic routines that help to transfer informa- 
tion between elements within the computer 100 such 
as during start-up, is stored in the ROM 140. The com- 
outer 1 00 also includes a hard disk drive 1 70 for readlnq 
» from and writing to a hard disk (not shown), a magnetic 
disk dnve 1 80 for reading from or writing to a removable 
magnetic disk 190, and an optical disk drive 191 for 
reading from or writing to a removable optical disk 192 
such as a CD ROM or other optical media. The hard disk 
dnve 1 70, magnetic disk drive 1 80, and optical disk drive 
1 91 are connected to the system bus 1 30 by a hard disk 
drive interface 1 92, a magnetic disk drive interface 1 93 
and an optical disk drive interface 1 94, respectively The 
drives and their associated computer-readable media 
provide nonvolatile storage of computer readable In- 
structions, data structures, program modules and other 
data for the personal computer 1 00. It will be appreciat- 
ed by those skilled in the art that othertypes of computer 
readable media that can store data that is accessible by 
a computer, such as magnetic cassettes, flash memory 
cards, digital video disks, Bernoulli cartridges, random 
access memories (RAMs) , read only memories (ROMs) 
and the like, may also be used in the example operatlna 
environment. 

» [0021] A number of program modules can be stored 
on the hard disk drive 170, magnetic disk 190, optical 
disk 192. ROM 140 or RAM 150, including an operating 
system 195, one or more application programs 196 oth- 
er program modules 1 97, and program data 1 98. Auser 
can enter commands and inf ormatio n into the computer 
100 through input devices such as a keyboard 101 and 
pointing device 102. Other input devices (not shown) 
may include a microphone, joystick, game pad, satellite 
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ceived signal as a user's input. Other methods of start- 
ing an operation or writing or contact with a digitizer are 
known in the art. For purposes of illustration and de- 
scription, the system and method reference physical 
contact with the digitizer. All other ways of providing sig- 
nals to a processor are considered within the scope of 
the invention and are not mentioned here for simplicity 
[0030] In step 302, the system determines the contact 
position and what lies beneath the contact position (for 
example, an object, a drawing, blank space, ink. and the 
like). In step 303, the system determines if the stylus 
has moved beyond a first threshold (time, distance, rate 
.. _or acceleration, and the like). In-one embodiment the 
threshold is set to the minimum resolvable movement 
In another embodiment, the threshold is set higher to 
account for shaky hands, vibrations of the digitizer or 
tablet pc (for example, if trying to use the system while 
driving in a car over a bumpy road). It is noted that ob- 
jects may have all the same threshold. Alternatively ob- 
jects may have different thresholds. This may be de- 
pendent on the object, the size of the object, the state 
of the system, the state of the object, and the like 
[0031J If the first threshold has been exceeded then 
the system proceeds to step 304 where the user's'input 

?n?f, $ !!i ed f aS 8 Str ° ke and ,he system ste P s t0 P^nt A 

305. If the first threshold has not been exceeded the 

system determines if the stylus was still in contact with 
the digitizer when a time threshold had expired in step 

306. If no (meaning that the stylus was still in contact 
with the digitizer surface), the system classifies the input 
as a tap in step 307 and proceeds to point B 308 
[0032] if the stylus was still in contact with the surface 
after the time threshold In step 306, the system deter- 
mines If a second move threshold was exceeded In step 
309. The first and second move thresholds may be iden- 
tical or different: For example, both may be .25 mm Or 
the first may be .5 mm or one mm and the second be ' 
3 mm. Further, the first may be 1 .2 mm or more and the 
second may be .5mm or more. In short, any values may 
be used as long as they are not obtrusive to the user 
The second threshold may be determined only after the 
time threshold of step 306 has expired. In this example 
the second threshold maybe higher than the first thresh- 
old (or it may be the same or smaller). 
[0033] If the second move threshold was not exceed- 
ed, then the system classifies the input as a hold in step 
310 and proceeds to point C 311. If the second move 
threshold was exceeded, then the system classifies the 
input as a 'hold and drag' in step 31 2 and moves to point 

[0034] Figure 4 shows point A as starting point 401 
Here, the system classified the input as a stroke and 
begins stroke processing in step 402. In step 403 the 
system determines if the stroke started on a draggable 
object. If yes, the system determines in step 404 wheth- 
er drag threshold was exceeded (for example, 25 
inches, .25 inches per second and the like) If so the 
system classifies the stroke as a drag in step 405and 



performs a function that is dependent on the object For 
example, the drag may extend a selection as described 
m greater detail in "Selection Handles in Editing Elec- 

5 T2T U T ntS: ' fi ' ed concurren % with the present 
application (attorney docket 03797.00069) and ex- 
pressly incorporated by reference. Also, the drag may 
operate a bungee tool as described in Serial No. (Ally 
docket 3797.00070), entitled "Insertion Point Bungee 
Space Tool", and filed concurrent with the present ap- 
plication, and expressly incorporated herein 
[0035] If, m step 404, the drag threshold has not been 
exceeded, the system maintains the current state (with 
the object being selected or not) in step 407. If the stroke 
was not over a draggable object in step 403, the system 
determines if the area under the contact point is inkable 
«n step 408. For example, inkable may mean an area 
capable of receiving ink (including drawings, annota- 
tions, or writing) as detailed in serial no. 60/212 825 
hied June 21 . 2000. and expressly incorporated herein 
by reference for essential subject matter. By contrast a 
control button (for copy, save, open, etc.) may not be 
inkable. If inkable in step 408, the system permits inking 
drawing, wnting, annotating and other related func 
ions) in step 409. If not inkable. the system maintains 

nnolT 6 " 1 St3te (0bjects selected 0f not) in step 407 
[0036] m Figure 5A, the system starts at point B 501 
and operates on the input as a tap 502. The system de- 
termines whether the tap was on an area or object that 

30 I 1 ",? 1316 in • 8 ! 8P 5 ° 3 ' ' f yes ' tne s * stem determines 
» whether any ink was recently added or "wet" (for exam- 
ple, less than .5 or 1 second old) in step 504. If so the 
system considers the tap as a dot to be added to the ink 
n step 505 (and adds the dot). If no wet ink exists, then 
the system determines if the tap was over a selectable 
object in step 506. It is noted that steps 503 and 504 
may be combined. If the tap was over a selectable ob- 
ject, then the system determines if the object was al- 
ready selected in step 507. If it was not, then the system 
selects the tapped object in step 508. If a previous object 
had been selected, the system cancels the previous or 
old selection in step 509. If the object was previously 
selected as determined by step 507, the system per- 
forms an action relevant to the object in step 510 This 
action may include editing the object, performing a pre- 
defined operation (for example, enlarge, shrink and the 
like). From step 506. if the tap was not on a selectable 
object, then the system proceeds to point BB 512 
[0037] Figure 5B shows additional processing to Fia- 
ure 5A. As point BB 512, the system determines if the 
tap was in a space between text (referred to herein as 
an inline space) in step 513. If y es . the system places 
an insertion point at the tap point in step 514. As shown 
in a broken lined box, the system may also cancel any 
old or previous selections in step 515. If no. then the 

2UT ? etermines if the ta P Point has ink nearby in step 
5 8. If the system determines that the tap was nearby 
ink, then the system adds a dot to the ink in step 51 6 If 
there was an old selection, then the system cancels the 
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time threshold and said Input does not satisfy said 
second move threshold, said input is classified as 
a hold. 

The method of classifying a user's input according 
to claim 2, wherein, if said input does notsalisfy said 
time threshold and said input satisfies said second 
move threshold, said input is classified as a hold 
and drag. 



9. A method of classifying a user's input to a computer 
comprising the steps of: 

receiving a user's input; 
classifying the user's input as one of a stroke, 
a tap, a hold, or a hold and drag based on at 
least one of the input satisfying a move thresh- 
old and a time threshold. 

10. A method of implementing a stroke input to a com- 
puter comprising the steps of: 

determining whether said stroke input started 
on a draggable object; 

determining whether said stroke input satisfies 
a drag threshold; 

in response to said first determining step and 
said second determining step, dragging said 
draggable object. 

11. A method of implementing a stroke input to a com- 
puter comprising the steps of: 

determining that said stroke input did not start 
on a draggable object; 

determining that location of said stroke input is 
inkable; and, 

adding ink to said location. 



simulating a right mouse click. 

15. The method of classifying a user's input according 
to claim 8, further comprising the step of: 

dragging a selected object. 

1 6. A computer-readable medium storing a program for 
operating a digitizer, said program comprising the 

10 steps of: 



receiving user input; 

classifying the user input by using at least one 
of a time threshold and a movement threshold; 
and, 

performing an action based on the user input. 



15 



1 7. A computer-readable medium storing a program for 
classifying a user's input to a computer, said pro- 
20 gram comprising the steps of: 



receiving a user's input; 

classifying the user's input as one of a stroke, 

a tap, a hold, or a hold and drag based on at 

least one of the input satisfying a first move 

threshold, a time threshold, and a second move 

threshold. 
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12. The method according to claim 11, wherein said 40 
adding ink step includes at least one of drawing, 
writing, or annotating. 

13. A method of implementing a tap input to a computer 
comprising the steps of: 45 

determining at least one of whether a location 
of said tap includes wet ink, whether said loca- 
tion is in an inline space, whether said location 
includes a selectable object, and whether said so 
object was previously selected; and 
performing at least one of adding a dot of ink, 
selecting said selectable object, placing an in- 
sertion point in said inline space, or performing 
an action associated with said object. 55 

14. The method of classifying a user's input according 
to claim 7, further comprising the step of: 



1 8. The computer-readable medium according to claim 
1 7, wherein said classifying step is based on at least 
two of the input satisfying said first move threshold, 
said time threshold, and said second move thresh- 
old. 

1 9. The computer-readable medium according to claim 
1 7, wherein said classifying step is based on the 
input satisfying said first move threshold, said time 
threshold, and said second move threshold. 

20. The computer-readable medium according to claim 
17, wherein, if said input satisfies said first move 
threshold, the input is classified as a stroke. 

21 . The computer-readable medium according to claim 
17, wherein, if said input does not satisfy said first 
move threshold and said input does not satisfy said 
time threshold, the input is classified as a tap. 

22. The computer-readable medium according to claim 
1 7, wherein, if said input does not satisfy said time 
threshold and said input does not satisfy said sec- 
ond move threshold, said input is classified as a 
hold. 

23. The computer-readable medium according to claim 
1 7, wherein, if said input does not satisfy said time 
threshold and said input satisfies said second move 
threshold, said input is classified as a hold and drag. 
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